{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Testing Custom Timeouts\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prerequistes\n",
    "You will need\n",
    " - [Git clone of Seldon Core](https://github.com/SeldonIO/seldon-core)\n",
    " - [Helm](https://github.com/kubernetes/helm)\n",
    " - [Minikube](https://github.com/kubernetes/minikube) version v0.24.0 or greater\n",
    " - [seldon-core Python package](https://pypi.org/project/seldon-core/) (```pip install seldon-core```)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create Cluster\n",
    "\n",
    "Start minikube and ensure custom resource validation is activated and there is 5G of memory. \n",
    "\n",
    "Your start command with the kvm driver would then look like:\n",
    "```\n",
    "minikube start --vm-driver kvm2 --memory 4096 --feature-gates=CustomResourceValidation=true --extra-config=apiserver.Authorization.Mode=RBAC\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "namespace/seldon created\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl create namespace seldon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Context \"minikube\" modified.\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl config set-context $(kubectl config current-context) --namespace=seldon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "clusterrolebinding.rbac.authorization.k8s.io/kube-system-cluster-admin created\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl create clusterrolebinding kube-system-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Install Helm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "serviceaccount/tiller created\n",
      "clusterrolebinding.rbac.authorization.k8s.io/tiller created\n",
      "$HELM_HOME has been configured at /home/janis/.helm.\n",
      "\n",
      "Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.\n",
      "\n",
      "Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.\n",
      "To prevent this, run `helm init` with the --tiller-tls-verify flag.\n",
      "For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation\n",
      "Happy Helming!\n"
     ]
    }
   ],
   "source": [
    "!kubectl -n kube-system create sa tiller\n",
    "!kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller\n",
    "!helm init --service-account tiller"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for deployment \"tiller-deploy\" rollout to finish: 0 of 1 updated replicas are available...\n",
      "deployment \"tiller-deploy\" successfully rolled out\n"
     ]
    }
   ],
   "source": [
    "!kubectl rollout status deploy/tiller-deploy -n kube-system"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Start seldon-core"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Install the custom resource definition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME:   seldon-core-crd\n",
      "LAST DEPLOYED: Tue Dec 18 11:10:48 2018\n",
      "NAMESPACE: seldon\n",
      "STATUS: DEPLOYED\n",
      "\n",
      "RESOURCES:\n",
      "==> v1beta1/Deployment\n",
      "NAME                        DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE\n",
      "seldon-spartakus-volunteer  1        0        0           0          0s\n",
      "\n",
      "==> v1/ServiceAccount\n",
      "NAME                        SECRETS  AGE\n",
      "seldon-spartakus-volunteer  1        0s\n",
      "\n",
      "==> v1beta1/ClusterRole\n",
      "NAME                        AGE\n",
      "seldon-spartakus-volunteer  0s\n",
      "\n",
      "==> v1beta1/ClusterRoleBinding\n",
      "NAME                        AGE\n",
      "seldon-spartakus-volunteer  0s\n",
      "\n",
      "==> v1/ConfigMap\n",
      "NAME                     DATA  AGE\n",
      "seldon-spartakus-config  3     0s\n",
      "\n",
      "==> v1beta1/CustomResourceDefinition\n",
      "NAME                                         AGE\n",
      "seldondeployments.machinelearning.seldon.io  0s\n",
      "\n",
      "\n",
      "NOTES:\n",
      "NOTES: TODO\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!helm install ../helm-charts/seldon-core-crd --name seldon-core-crd --set usage_metrics.enabled=true"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME:   seldon-core\n",
      "LAST DEPLOYED: Tue Dec 18 11:10:51 2018\n",
      "NAMESPACE: seldon\n",
      "STATUS: DEPLOYED\n",
      "\n",
      "RESOURCES:\n",
      "==> v1/Service\n",
      "NAME                          TYPE       CLUSTER-IP      EXTERNAL-IP  PORT(S)                        AGE\n",
      "seldon-core-ambassador-admin  NodePort   10.108.119.46   <none>       8877:30383/TCP                 1s\n",
      "seldon-core-ambassador        NodePort   10.110.98.62    <none>       80:31892/TCP,443:32336/TCP     1s\n",
      "seldon-core-seldon-apiserver  NodePort   10.105.205.239  <none>       8080:30495/TCP,5000:30228/TCP  0s\n",
      "seldon-core-redis             ClusterIP  10.107.101.105  <none>       6379/TCP                       0s\n",
      "\n",
      "==> v1beta1/Deployment\n",
      "NAME                                DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE\n",
      "seldon-core-ambassador              1        1        1           0          0s\n",
      "seldon-core-seldon-apiserver        1        1        1           0          0s\n",
      "seldon-core-seldon-cluster-manager  1        1        1           0          0s\n",
      "seldon-core-redis                   1        1        1           0          0s\n",
      "\n",
      "==> v1/Pod(related)\n",
      "NAME                                                 READY  STATUS             RESTARTS  AGE\n",
      "seldon-core-ambassador-56cb8fc595-nrxnw              0/1    ContainerCreating  0         0s\n",
      "seldon-core-seldon-apiserver-59978fbf45-cd9sr        0/1    ContainerCreating  0         0s\n",
      "seldon-core-seldon-cluster-manager-5449b8dd5f-689h9  0/1    ContainerCreating  0         0s\n",
      "seldon-core-redis-567db9ddcf-dwvbp                   0/1    ContainerCreating  0         0s\n",
      "\n",
      "==> v1/ServiceAccount\n",
      "NAME    SECRETS  AGE\n",
      "seldon  1        1s\n",
      "\n",
      "==> v1beta1/Role\n",
      "NAME          AGE\n",
      "seldon-local  1s\n",
      "ambassador    1s\n",
      "\n",
      "==> v1beta1/RoleBinding\n",
      "NAME        AGE\n",
      "ambassador  1s\n",
      "\n",
      "==> v1/RoleBinding\n",
      "NAME    AGE\n",
      "seldon  1s\n",
      "\n",
      "\n",
      "NOTES:\n",
      "Thank you for installing Seldon Core.\n",
      "\n",
      "Documentation can be found at https://github.com/SeldonIO/seldon-core\n",
      "\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!helm install ../helm-charts/seldon-core --name seldon-core --namespace seldon \\\n",
    "    --set ambassador.enabled=true"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check all services are running before proceeding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for deployment \"seldon-core-seldon-cluster-manager\" rollout to finish: 0 of 1 updated replicas are available...\n",
      "deployment \"seldon-core-seldon-cluster-manager\" successfully rolled out\n",
      "Waiting for deployment \"seldon-core-seldon-apiserver\" rollout to finish: 0 of 1 updated replicas are available...\n",
      "deployment \"seldon-core-seldon-apiserver\" successfully rolled out\n",
      "Waiting for deployment \"seldon-core-ambassador\" rollout to finish: 0 of 1 updated replicas are available...\n",
      "deployment \"seldon-core-ambassador\" successfully rolled out\n"
     ]
    }
   ],
   "source": [
    "!kubectl rollout status deploy/seldon-core-seldon-cluster-manager\n",
    "!kubectl rollout status deploy/seldon-core-seldon-apiserver\n",
    "!kubectl rollout status deploy/seldon-core-ambassador "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set up REST and gRPC methods\n",
    "\n",
    "**Ensure you port forward to API Gateway**\n",
    "\n",
    "REST:\n",
    "\n",
    "```\n",
    "kubectl port-forward $(kubectl get pods -n seldon -l app=seldon-apiserver-container-app -o jsonpath='{.items[0].metadata.name}') -n seldon 8003:8080\n",
    "```\n",
    "\n",
    "GRPC:\n",
    "\n",
    "```\n",
    "kubectl port-forward $(kubectl get pods -n seldon -l app=seldon-apiserver-container-app -o jsonpath='{.items[0].metadata.name}') -n seldon 8004:5000\n",
    "```\n",
    "\n",
    "**Ensure you port forward ambassador**:\n",
    "\n",
    "```\n",
    "kubectl port-forward $(kubectl get pods -n seldon -l service=ambassador -o jsonpath='{.items[0].metadata.name}') -n seldon 8005:8080\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Illustration of both REST and gRPC requests. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from requests.auth import HTTPBasicAuth\n",
    "from seldon_core.proto import prediction_pb2\n",
    "from seldon_core.proto import prediction_pb2_grpc\n",
    "import grpc\n",
    "import numpy as np\n",
    "\n",
    "AMBASSADOR_API=\"localhost:8005\"\n",
    "GATEWAY_REST=\"localhost:8003\"\n",
    "GATEWAY_GRPC=\"localhost:8004\"\n",
    "\n",
    "def get_token(oauth_key,oauth_secret):\n",
    "    payload = {'grant_type': 'client_credentials'}\n",
    "    response = requests.post(\n",
    "                \"http://\"+GATEWAY_REST+\"/oauth/token\",\n",
    "                auth=HTTPBasicAuth(oauth_key, oauth_secret),\n",
    "                data=payload)\n",
    "    print(response.text)\n",
    "    token =  response.json()[\"access_token\"]\n",
    "    return token\n",
    "\n",
    "    \n",
    "def rest_request_api_gateway(oauth_key,oauth_secret):\n",
    "    token = get_token(oauth_key,oauth_secret)\n",
    "    headers = {'Authorization': 'Bearer '+token}\n",
    "    payload = {\"data\":{\"names\":[\"a\",\"b\"],\"tensor\":{\"shape\":[2,2],\"values\":[0,0,1,1]}}}\n",
    "    response = requests.post(\n",
    "                \"http://\"+GATEWAY_REST+\"/api/v0.1/predictions\",\n",
    "                headers=headers,\n",
    "                json=payload)\n",
    "    print(response.text)\n",
    "\n",
    "def grpc_request_api_gateway(oauth_key,oauth_secret,data_size):\n",
    "    token = get_token(oauth_key,oauth_secret)\n",
    "    shape = [1,data_size]\n",
    "    arr = np.random.rand(data_size)\n",
    "    datadef = prediction_pb2.DefaultData(\n",
    "            names = [\"a\",\"b\"],\n",
    "            tensor = prediction_pb2.Tensor(\n",
    "                shape = shape,\n",
    "                values = arr\n",
    "                )\n",
    "            )\n",
    "    request = prediction_pb2.SeldonMessage(data = datadef)\n",
    "    channel = grpc.insecure_channel(GATEWAY_GRPC)\n",
    "    stub = prediction_pb2_grpc.SeldonStub(channel)\n",
    "    metadata = [('oauth_token', token)]\n",
    "    response = stub.Predict(request=request,metadata=metadata)\n",
    "    print(response)\n",
    "    \n",
    "def grpc_request_ambassador(deploymentName,data_size):\n",
    "    shape = [1,data_size]\n",
    "    arr = np.random.rand(data_size)\n",
    "    datadef = prediction_pb2.DefaultData(\n",
    "            names = [\"a\",\"b\"],\n",
    "            tensor = prediction_pb2.Tensor(\n",
    "                shape = shape,\n",
    "                values = arr\n",
    "                )\n",
    "            )\n",
    "    request = prediction_pb2.SeldonMessage(data = datadef)\n",
    "    channel = grpc.insecure_channel(AMBASSADOR_API)\n",
    "    stub = prediction_pb2_grpc.SeldonStub(channel)\n",
    "    metadata = [('seldon',deploymentName)]\n",
    "    response = stub.Predict(request=request,metadata=metadata)\n",
    "    print(response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "    \u001b[34;01m\"apiVersion\"\u001b[39;49;00m: \u001b[33m\"machinelearning.seldon.io/v1alpha2\"\u001b[39;49;00m,\r\n",
      "    \u001b[34;01m\"kind\"\u001b[39;49;00m: \u001b[33m\"SeldonDeployment\"\u001b[39;49;00m,\r\n",
      "    \u001b[34;01m\"metadata\"\u001b[39;49;00m: {\r\n",
      "        \u001b[34;01m\"labels\"\u001b[39;49;00m: {\r\n",
      "            \u001b[34;01m\"app\"\u001b[39;49;00m: \u001b[33m\"seldon\"\u001b[39;49;00m\r\n",
      "        },\r\n",
      "        \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"seldon-deployment-example\"\u001b[39;49;00m\r\n",
      "    },\r\n",
      "    \u001b[34;01m\"spec\"\u001b[39;49;00m: {\r\n",
      "        \u001b[34;01m\"annotations\"\u001b[39;49;00m: {\r\n",
      "            \u001b[34;01m\"project_name\"\u001b[39;49;00m: \u001b[33m\"FX Market Prediction\"\u001b[39;49;00m,\r\n",
      "            \u001b[34;01m\"deployment_version\"\u001b[39;49;00m: \u001b[33m\"v1\"\u001b[39;49;00m,\r\n",
      "\t    \u001b[34;01m\"seldon.io/rest-read-timeout\"\u001b[39;49;00m:\u001b[33m\"1\"\u001b[39;49;00m\r\n",
      "        },\r\n",
      "        \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"test-deployment\"\u001b[39;49;00m,\r\n",
      "        \u001b[34;01m\"oauth_key\"\u001b[39;49;00m: \u001b[33m\"oauth-key\"\u001b[39;49;00m,\r\n",
      "        \u001b[34;01m\"oauth_secret\"\u001b[39;49;00m: \u001b[33m\"oauth-secret\"\u001b[39;49;00m,\r\n",
      "        \u001b[34;01m\"predictors\"\u001b[39;49;00m: [\r\n",
      "            {\r\n",
      "                \u001b[34;01m\"componentSpecs\"\u001b[39;49;00m: [{\r\n",
      "                    \u001b[34;01m\"spec\"\u001b[39;49;00m: {\r\n",
      "                        \u001b[34;01m\"containers\"\u001b[39;49;00m: [\r\n",
      "                            {\r\n",
      "                                \u001b[34;01m\"image\"\u001b[39;49;00m: \u001b[33m\"seldonio/mock_classifier:1.0\"\u001b[39;49;00m,\r\n",
      "                                \u001b[34;01m\"imagePullPolicy\"\u001b[39;49;00m: \u001b[33m\"IfNotPresent\"\u001b[39;49;00m,\r\n",
      "                                \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"classifier\"\u001b[39;49;00m,\r\n",
      "                                \u001b[34;01m\"resources\"\u001b[39;49;00m: {\r\n",
      "                                    \u001b[34;01m\"requests\"\u001b[39;49;00m: {\r\n",
      "                                        \u001b[34;01m\"memory\"\u001b[39;49;00m: \u001b[33m\"1Mi\"\u001b[39;49;00m\r\n",
      "                                    }\r\n",
      "                                }\r\n",
      "                            }\r\n",
      "                        ],\r\n",
      "                        \u001b[34;01m\"terminationGracePeriodSeconds\"\u001b[39;49;00m: \u001b[34m20\u001b[39;49;00m\r\n",
      "                    }\r\n",
      "                }],\r\n",
      "                \u001b[34;01m\"graph\"\u001b[39;49;00m: {\r\n",
      "                    \u001b[34;01m\"children\"\u001b[39;49;00m: [],\r\n",
      "                    \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"classifier\"\u001b[39;49;00m,\r\n",
      "                    \u001b[34;01m\"endpoint\"\u001b[39;49;00m: {\r\n",
      "\t\t\t\u001b[34;01m\"type\"\u001b[39;49;00m : \u001b[33m\"REST\"\u001b[39;49;00m\r\n",
      "\t\t    },\r\n",
      "                    \u001b[34;01m\"type\"\u001b[39;49;00m: \u001b[33m\"MODEL\"\u001b[39;49;00m\r\n",
      "                },\r\n",
      "                \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"fx-market-predictor\"\u001b[39;49;00m,\r\n",
      "                \u001b[34;01m\"replicas\"\u001b[39;49;00m: \u001b[34m1\u001b[39;49;00m,\r\n",
      "\t\t\u001b[34;01m\"annotations\"\u001b[39;49;00m: {\r\n",
      "\t\t    \u001b[34;01m\"predictor_version\"\u001b[39;49;00m : \u001b[33m\"v1\"\u001b[39;49;00m\r\n",
      "\t\t}\r\n",
      "            }\r\n",
      "        ]\r\n",
      "    }\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!pygmentize resources/model_short_timeouts.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Seldon Deployment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Deploy the runtime graph to kubernetes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io/seldon-deployment-example created\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl apply -f resources/model_short_timeouts.json -n seldon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME                        CREATED AT\r\n",
      "seldon-deployment-example   2s\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl get seldondeployments -n seldon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name:         seldon-deployment-example\r\n",
      "Namespace:    seldon\r\n",
      "Labels:       app=seldon\r\n",
      "Annotations:  kubectl.kubernetes.io/last-applied-configuration:\r\n",
      "                {\"apiVersion\":\"machinelearning.seldon.io/v1alpha2\",\"kind\":\"SeldonDeployment\",\"metadata\":{\"name\":\"seldon-deployment-example\",\"namespace\":\"s...\r\n",
      "API Version:  machinelearning.seldon.io/v1alpha2\r\n",
      "Kind:         SeldonDeployment\r\n",
      "Metadata:\r\n",
      "  Cluster Name:        \r\n",
      "  Creation Timestamp:  2018-12-18T11:12:01Z\r\n",
      "  Generation:          1\r\n",
      "  Resource Version:    746\r\n",
      "  Self Link:           /apis/machinelearning.seldon.io/v1alpha2/namespaces/seldon/seldondeployments/seldon-deployment-example\r\n",
      "  UID:                 be7a2bce-02b5-11e9-aed1-18941d3b439b\r\n",
      "Spec:\r\n",
      "  Annotations:\r\n",
      "    Deployment Version:                   v1\r\n",
      "    Project Name:                         FX Market Prediction\r\n",
      "    Seldon . Io / Rest - Read - Timeout:  1\r\n",
      "  Name:                                   test-deployment\r\n",
      "  Oauth Key:                              oauth-key\r\n",
      "  Oauth Secret:                           oauth-secret\r\n",
      "  Predictors:\r\n",
      "    Annotations:\r\n",
      "      Predictor Version:  v1\r\n",
      "    Component Specs:\r\n",
      "      Spec:\r\n",
      "        Containers:\r\n",
      "          Image:              seldonio/mock_classifier:1.0\r\n",
      "          Image Pull Policy:  IfNotPresent\r\n",
      "          Name:               classifier\r\n",
      "          Resources:\r\n",
      "            Requests:\r\n",
      "              Memory:                      1Mi\r\n",
      "        Termination Grace Period Seconds:  20\r\n",
      "    Graph:\r\n",
      "      Endpoint:\r\n",
      "        Type:  REST\r\n",
      "      Name:    classifier\r\n",
      "      Type:    MODEL\r\n",
      "    Name:      fx-market-predictor\r\n",
      "    Replicas:  1\r\n",
      "Status:\r\n",
      "  State:  Creating\r\n",
      "Events:   <none>\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl describe seldondeployments seldon-deployment-example -n seldon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the status of the SeldonDeployment. **When ready the replicasAvailable should be 1**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "map[predictorStatus:[map[replicasAvailable:1 name:test-deployment-fx-market-predictor-7cd068f replicas:1]] state:Available]"
     ]
    }
   ],
   "source": [
    "!kubectl get seldondeployments seldon-deployment-example -o jsonpath='{.status}' -n seldon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get predictions with short and long timeouts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This next request should fail as the timeout is too short"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"access_token\":\"7bdc8db7-e6a9-4a4a-80c9-c208992bff1b\",\"token_type\":\"bearer\",\"expires_in\":43199,\"scope\":\"read write\"}\n",
      "{\n",
      "  \"code\": 103,\n",
      "  \"info\": \"Status code: 500 Reason: \",\n",
      "  \"reason\": \"Microservice error\",\n",
      "  \"status\": \"FAILURE\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "rest_request_api_gateway('oauth-key','oauth-secret')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Delete this graph and recreate one with a longer timeout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io \"seldon-deployment-example\" deleted\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl delete -f resources/model_short_timeouts.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "    \u001b[34;01m\"apiVersion\"\u001b[39;49;00m: \u001b[33m\"machinelearning.seldon.io/v1alpha2\"\u001b[39;49;00m,\r\n",
      "    \u001b[34;01m\"kind\"\u001b[39;49;00m: \u001b[33m\"SeldonDeployment\"\u001b[39;49;00m,\r\n",
      "    \u001b[34;01m\"metadata\"\u001b[39;49;00m: {\r\n",
      "        \u001b[34;01m\"labels\"\u001b[39;49;00m: {\r\n",
      "            \u001b[34;01m\"app\"\u001b[39;49;00m: \u001b[33m\"seldon\"\u001b[39;49;00m\r\n",
      "        },\r\n",
      "        \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"seldon-deployment-example\"\u001b[39;49;00m\r\n",
      "    },\r\n",
      "    \u001b[34;01m\"spec\"\u001b[39;49;00m: {\r\n",
      "        \u001b[34;01m\"annotations\"\u001b[39;49;00m: {\r\n",
      "            \u001b[34;01m\"project_name\"\u001b[39;49;00m: \u001b[33m\"FX Market Prediction\"\u001b[39;49;00m,\r\n",
      "            \u001b[34;01m\"deployment_version\"\u001b[39;49;00m: \u001b[33m\"v1\"\u001b[39;49;00m,\r\n",
      "\t    \u001b[34;01m\"seldon.io/rest-read-timeout\"\u001b[39;49;00m:\u001b[33m\"10000\"\u001b[39;49;00m,\r\n",
      "\t    \u001b[34;01m\"seldon.io/rest-connection-timeout\"\u001b[39;49;00m:\u001b[33m\"10000\"\u001b[39;49;00m,\t    \r\n",
      "\t    \u001b[34;01m\"seldon.io/grpc-read-timeout\"\u001b[39;49;00m:\u001b[33m\"10000\"\u001b[39;49;00m\r\n",
      "        },\r\n",
      "        \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"test-deployment\"\u001b[39;49;00m,\r\n",
      "        \u001b[34;01m\"oauth_key\"\u001b[39;49;00m: \u001b[33m\"oauth-key\"\u001b[39;49;00m,\r\n",
      "        \u001b[34;01m\"oauth_secret\"\u001b[39;49;00m: \u001b[33m\"oauth-secret\"\u001b[39;49;00m,\r\n",
      "        \u001b[34;01m\"predictors\"\u001b[39;49;00m: [\r\n",
      "            {\r\n",
      "                \u001b[34;01m\"componentSpecs\"\u001b[39;49;00m: [{\r\n",
      "                    \u001b[34;01m\"spec\"\u001b[39;49;00m: {\r\n",
      "                        \u001b[34;01m\"containers\"\u001b[39;49;00m: [\r\n",
      "                            {\r\n",
      "                                \u001b[34;01m\"image\"\u001b[39;49;00m: \u001b[33m\"seldonio/mock_classifier:1.0\"\u001b[39;49;00m,\r\n",
      "                                \u001b[34;01m\"imagePullPolicy\"\u001b[39;49;00m: \u001b[33m\"IfNotPresent\"\u001b[39;49;00m,\r\n",
      "                                \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"classifier\"\u001b[39;49;00m,\r\n",
      "                                \u001b[34;01m\"resources\"\u001b[39;49;00m: {\r\n",
      "                                    \u001b[34;01m\"requests\"\u001b[39;49;00m: {\r\n",
      "                                        \u001b[34;01m\"memory\"\u001b[39;49;00m: \u001b[33m\"1Mi\"\u001b[39;49;00m\r\n",
      "                                    }\r\n",
      "                                }\r\n",
      "                            }\r\n",
      "                        ],\r\n",
      "                        \u001b[34;01m\"terminationGracePeriodSeconds\"\u001b[39;49;00m: \u001b[34m20\u001b[39;49;00m\r\n",
      "                    }\r\n",
      "                }],\r\n",
      "                \u001b[34;01m\"graph\"\u001b[39;49;00m: {\r\n",
      "                    \u001b[34;01m\"children\"\u001b[39;49;00m: [],\r\n",
      "                    \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"classifier\"\u001b[39;49;00m,\r\n",
      "                    \u001b[34;01m\"endpoint\"\u001b[39;49;00m: {\r\n",
      "\t\t\t\u001b[34;01m\"type\"\u001b[39;49;00m : \u001b[33m\"REST\"\u001b[39;49;00m\r\n",
      "\t\t    },\r\n",
      "                    \u001b[34;01m\"type\"\u001b[39;49;00m: \u001b[33m\"MODEL\"\u001b[39;49;00m\r\n",
      "                },\r\n",
      "                \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"fx-market-predictor\"\u001b[39;49;00m,\r\n",
      "                \u001b[34;01m\"replicas\"\u001b[39;49;00m: \u001b[34m1\u001b[39;49;00m,\r\n",
      "\t\t\u001b[34;01m\"annotations\"\u001b[39;49;00m: {\r\n",
      "\t\t    \u001b[34;01m\"predictor_version\"\u001b[39;49;00m : \u001b[33m\"v1\"\u001b[39;49;00m\r\n",
      "\t\t}\r\n",
      "            }\r\n",
      "        ]\r\n",
      "    }\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!pygmentize resources/model_long_timeouts.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io/seldon-deployment-example created\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl apply -f resources/model_long_timeouts.json -n seldon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "map[state:Available predictorStatus:[map[name:test-deployment-fx-market-predictor-7cd068f replicas:1 replicasAvailable:1]]]"
     ]
    }
   ],
   "source": [
    "!kubectl get seldondeployments seldon-deployment-example -o jsonpath='{.status}' -n seldon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This next request should work as the timeout is much longer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"access_token\":\"7bdc8db7-e6a9-4a4a-80c9-c208992bff1b\",\"token_type\":\"bearer\",\"expires_in\":43126,\"scope\":\"read write\"}\n",
      "{\n",
      "  \"meta\": {\n",
      "    \"puid\": \"8jq85h9pbdquf9uii6nh6majr7\",\n",
      "    \"tags\": {\n",
      "    },\n",
      "    \"routing\": {\n",
      "    },\n",
      "    \"requestPath\": {\n",
      "      \"classifier\": \"seldonio/mock_classifier:1.0\"\n",
      "    },\n",
      "    \"metrics\": []\n",
      "  },\n",
      "  \"data\": {\n",
      "    \"names\": [\"proba\"],\n",
      "    \"tensor\": {\n",
      "      \"shape\": [2, 1],\n",
      "      \"values\": [0.05133579311531625, 0.12823373759251927]\n",
      "    }\n",
      "  }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "rest_request_api_gateway('oauth-key','oauth-secret')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
